fix(speculate): write the merge state before dispatching the batch - #585
Merged
Merged
Conversation
behinddwalls
marked this pull request as ready for review
August 13, 2026 18:41
behinddwalls
force-pushed
the
preetam/speculate-write-before-dispatch
branch
from
August 13, 2026 18:42
ea69c25 to
bcea46e
Compare
## Summary ### Why? `applyOutcome` published a batch to the merge topic before writing `BatchStateMerging`, so a lost compare-and-swap could leave Runway acting on an outcome that was never recorded. That ordering existed to avoid a stall, and the stall is real: nothing re-drives a batch stuck in `Merging`. `Process` self-heals only terminal and `Created` batches, `finalize` walks only heads that are still speculating, and the sole production reader of `BatchStateMerging` is the cancel controller — so a batch written `Merging` whose dispatch never went out would sit there forever. Giving that stall a repair path lets the write come first, which is the ordering the rest of the state machine already wants. ### What? `applyOutcome` is restructured into decide-state → recover → write → dispatch. The `terminal` bool falls out: a second switch mirrors the first and dispatches merge or conclude once the state write has landed. `recoverable` is hoisted above the switch so a cascade-decided *merge* gets a recovery message too, not just a cascade-decided failure. It needs one for the same reason: the write drops the batch out of the speculating set, and `Process`'s self-heal only ever names the trigger batch. `Process` gains a `BatchStateMerging` branch that re-sends the dispatch through the new `dispatchMerge` helper. That keeps the stable `IntentID`, the inverse of `fanout`'s `UniqueID` — for conclude a stable ID would suppress the repair, for merge it is what stops Runway merging the batch twice. One side benefit: a lost state CAS now means the dispatch is never sent at all, narrowing the window where a cancelled batch has a live merge request against it. ## Test Plan ✅ `bazel test //submitqueue/... //platform/...` — 68 tests pass New coverage: the dispatch follows the state write; a lost CAS publishes nothing; a cascade-merged batch gets its recovery signal before the write; `Process` on a `Merging` batch re-dispatches; `dispatchMerge` reuses one message ID per batch. `TestProcess_MergingRunsButDoesNotAct` asserted the old behaviour — that a `Merging` batch publishes nothing — and is replaced by `TestProcess_MergingSelfHeals`.
behinddwalls
force-pushed
the
preetam/speculate-write-before-dispatch
branch
from
August 13, 2026 19:06
bcea46e to
854d9b2
Compare
mnoah1
approved these changes
Aug 13, 2026
manjari25
approved these changes
Aug 13, 2026
behinddwalls
added a commit
to behinddwalls/submitqueue
that referenced
this pull request
Aug 19, 2026
uber#586) ## Summary ### Why? A request's trail read `batched → speculating → speculated → speculating → speculated → landing → landed`, and the repeats looked like the pipeline regressing. They were not a reporting glitch: `RequestStatusSpeculated` meant "a build passed on a path still consistent with how its dependencies are resolving", so it was published while the batch was still blocked, and `reportSpeculation` republished `speculating` whenever a dependency later resolved against that path's guess. Each extra pair was one speculative guess that passed and was then invalidated. That made `speculated` a per-path, provisional fact wearing a status — the exact shape `RequestEvent` exists for. A batch is not done speculating until it can be sent to merge; waiting on dependencies is still speculating. ### What? Two events join the vocabulary. `waiting` records that a path passed and the batch has nothing of its own left to run; `invalidated` records that a dependency resolved against the guess that path made. Both are occurrence-keyed on the path ID, so a passed path re-observed across runs collapses to one entry. `waiting` is gated on `outcomeWait` rather than on merely holding a live passed path. A merge is decided on that same predicate — `mergeablePath` implies `livePassedPath` — so an ungated report would claim a wait on every request that merges straight through. `reportSpeculation` moves below `decide` to see the outcome; both it and `decide` only read, so the reorder observes nothing different. `speculated` stays a status but now means speculation finished, published from `dispatchMerge` once the batch is cleared to merge. It goes ahead of the dispatch because the merge stage publishes `landing` as its first act on receiving one, and both statuses are non-terminal — so a `speculated` sent afterwards could carry the later timestamp and beat `landing` in the summary. The `hadPassed && !hasPassed` republish of `speculating` is gone. The status never leaves, so there is nothing to republish, and the oscillation goes with it. One trade-off worth naming: `speculated` is now near-instantaneous, so "is this batch blocked on dependencies?" is answerable from the latest event rather than from the status. The second commit adds the end-to-end coverage this had been missing. Nothing in `test/integration/` touches the speculate pipeline — the orchestrator integration suite is `TestPingAPI` and nothing else — and the e2e happy path has no dependencies, so it never speculates across one. `e2e-respeculate-queue` is registered in the gateway's queue list and deliberately takes no profile of its own: falling through to the baseline is what gives it the `all` analyzer, which serializes the queue so a second request becomes a batch depending on the first. The new test forces the wait rather than racing it. Batch IDs come from a per-queue counter as `<queue>/batch/<n>`, so on a fresh queue the leader is `batch/1`, and the build topic partitions by batch — closing the consumer gate on that partition before anything is published holds the leader's build and nothing else. The follower reaches a passed path while its dependency is still outstanding, reports `waiting`, and is asserted to still be `speculating`. Releasing the gate fails the leader, and the follower re-plans and lands with `speculating` and `speculated` recorded exactly once each. `invalidated` is deliberately not asserted end to end. A passed path stops occupying build budget, so by the time the leader fails the follower has usually funded the other side of the guess as well; it never loses its last live passed path, which is the state `invalidated` reports. Forcing that end to end would mean starving the queue's budget, which cannot be done without also starving the follower's first build. A unit test covers the case e2e cannot reach deterministically: a dependency turning terminal in the same run that walks the head resting on it, so the break is seen by a later generation of the finalize loop rather than by the read. A third commit adapts `TestDependentBatch_IsWokenByTheMergeAhead`, which uber#576 added to main after this branch was cut. It parks the lead batch's merge behind a closed gate, waits for the dependent to reach `speculated`, and only then opens the gate — an ordering that encodes the old meaning of `speculated`, which a batch could reach while its dependency was still outstanding. Under this change the dependent rests at `speculating` instead, so the test waited for a status that could not arrive until it opened the gate, and did not open the gate until it arrived. It now waits for the `waiting` event: the same fact it was reaching for, expressed as the signal that carries it, and reachable while the lead is still parked. The gate still opens next and both requests are still asserted to land, so the wake-up remains attributable to the fan-out alone. ## Test Plan ✅ `bazel test //submitqueue/... //platform/... //service/...` — 73 tests pass ✅ `bazel test //test/e2e/...` — 3/3 pass, including the new scenario ✅ `make lint`, `make check-gazelle`, `make check-tidy`, `make check-mocks` Re-verified after rebasing onto main at `42d1cb72`: `bazel test //submitqueue/... //platform/...` — 69 tests pass; `bazel test //test/e2e/submitqueue` — passes in 120s, against the 300s timeout it hit before the third commit. The two `reportSpeculation` tests now assert events. New unit coverage: a merging head reports `speculated` and no wait (the gate's regression test); `speculated` is published before the `TopicKeyMerge` dispatch; and the same-run cascade reports `invalidated`. `TestLand_HappyPath_ReachesLanded` needs no change: `speculating → speculated → landing → landed` still holds as an ordered subsequence, now for a different reason and at a different point in time. ## Issue Closes https://linear.app/uber/issue/CODEM-443 ## Stack 1. uber#585 1. @ uber#586
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why?
applyOutcomepublished a batch to the merge topic before writingBatchStateMerging, so a lost compare-and-swap could leave Runway acting on an outcome that was never recorded. That ordering existed to avoid a stall, and the stall is real: nothing re-drives a batch stuck inMerging.Processself-heals only terminal andCreatedbatches,finalizewalks only heads that are still speculating, and the sole production reader ofBatchStateMergingis the cancel controller — so a batch writtenMergingwhose dispatch never went out would sit there forever.Giving that stall a repair path lets the write come first, which is the ordering the rest of the state machine already wants.
What?
applyOutcomeis restructured into decide-state → recover → write → dispatch. Theterminalbool falls out: a second switch mirrors the first and dispatches merge or conclude once the state write has landed.recoverableis hoisted above the switch so a cascade-decided merge gets a recovery message too, not just a cascade-decided failure. It needs one for the same reason: the write drops the batch out of the speculating set, andProcess's self-heal only ever names the trigger batch.Processgains aBatchStateMergingbranch that re-sends the dispatch through the newdispatchMergehelper. That keeps the stableIntentID, the inverse offanout'sUniqueID— for conclude a stable ID would suppress the repair, for merge it is what stops Runway merging the batch twice.One side benefit: a lost state CAS now means the dispatch is never sent at all, narrowing the window where a cancelled batch has a live merge request against it.
Test Plan
✅
bazel test //submitqueue/... //platform/...— 68 tests passNew coverage: the dispatch follows the state write; a lost CAS publishes nothing; a cascade-merged batch gets its recovery signal before the write;
Processon aMergingbatch re-dispatches;dispatchMergereuses one message ID per batch.TestProcess_MergingRunsButDoesNotActasserted the old behaviour — that aMergingbatch publishes nothing — and is replaced byTestProcess_MergingSelfHeals.Stack